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One conception of teacher evaluation treats an 
evaluative Judgment (E) as a result of estimating the 
discrepancy between the instructor ^ £5 observed frequency of 
behavior ( 0 ) and the ideal frequency ( I ) . In a s i ngle 
questionnaire 5 students reported their ideal frequency of 
occurrence (l-scores) for common teach inc behaviors, the 
observed frequency of occurrence (O-scores), their evalua- 
tion on an "outstanding'' to "poor" scale for each behavior 
expressed as a^ teaching attribute (E-score), and their 
overall evaluation of the instructor. A discrepancy score 
(D-score) was computed for each student as the absolute 
difference between the I-score and 0-score foi; each be- 
havior. 

The data showed that I-score responses were variable 
among student respondents, but that the distribution of I- 
scores did not substantially vary with respect to either 

course content, course leveli sex of student, anticipated 
grade or overall evaluation of the teacher. 

Thf? discrepancy liypothesis would predict signifi- 
cant D-score v£. E-score correlations in the overall popu- 
lation and also for each I- score subpopulation (defined by 
their I-score response toward a given behavior )• Correla- 
tions were significant for only those having an I-score at 
an extreme end of the frequency scale, suggesting tha*; a 
strict discrepancy process is not being used by all student 
raters* Alternative hypotheses are discussed. 
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Overall objective ; The fundamental question of 
this., project has been the validity and usefulness of the idea 
that teacher evaluation is the estimate of discrepancy fbetween 
ideal behavior and real (observed) behavior in an educational 
setting, e.nd v-hat teacher evaluation procedures should incor- 
porate such a discrepancy analysis. 

B. Terminologry ; Although a more detailed diccuccion 
is found in the Proposal (Pp. 12-llt), a brief description of 
the variables studied in this project follows, for the con- 
venience of the reader. 

1) I-6cores . For each beaching behavior, what fre- 
quency ("almost always," "often," "occasionally," "seldom," 
"almost never") of occurrence does the evaluator (student) 
prefer? 

2) O-scores . For each teaching behavior, what fre- 
quency ('almost always," "ofteu," etc.) of occurrence has the 
evaluator (student) observed? 

3) E-scores . For each teaching attribute, what 
evaluative assessment ("Outstanding," "good," "adequate," 
"fair," "poor") would fit the teacher? 

h) D -scores . The computed absolute difference 
between the I-score and the 0-score for each behavior. 

5) S-scores . For each teaching attribute, how 
important (salient) would an evaluation on that attribute be 
on an overall evaluation of the teacher ("Of next to no impor- 
tance," "Not very important," "Of only moderate importance," 
"Important," "Extremely important"). 

6) ET-score s . What overall assessment of teach- 
ing effectiveness could be made ("outstanding,," "good," etc.) 
.)f the teacher. 

0. S ome Core Questions ; Although the outcome of the 
project will be presented in roughly chronological form, there 
is % cluster of fundamental issues that will be analysed in 
more than one of the studies. They are presented here and 
will be discussed later as a whole in the Conclusions section. 

l) What are the characteristics of I-scores? Are 
they variable (i.e., do they reveal a lack of unanimity re- 



gardine, ideal?) and » if thoy nro, in the distribution of 1- 
sooree in a student arimple stable over timo, over differencea 
ir* subject matter, subject level, anu studsiit cliaj*^acteristiGQ? 

2) Are individual item responses related to an 
estimation of overall teaching ability? Specifically are 
I-scores, 0-scores, K-scores and D~scores correlated to 
ET-scores? Are they independent? 

3) How is the D-score related to the E-score? Is 
the 1) vs. E association closer than when one relates the more 
traditional 0-score with the E-score? 

Does a computed aggregate of E-scores (such as 
an arithmetic mean) or a similarly computed aggregate of D- 
scores ■ approximate the E,p-score better than a single item? 
Does it help to weigh each item response by the appropriate 
S-scor e? 

^ ^ ' Procedure and results of a preliminary study (code: 

1) R ationale t The intention of this preliminary 
study was to test out a collection of 30 teaching behaviors 
(see Appendix B) for clarity and reasonableness, and to obtain 
a collection of I-scores. 

2) Sample characteristics ; The student sample con- 
sisted of i+T Hofstra University undergraduate biology-major 
seniors reacting in one classroom to a "single instructor. The 
course involved was of a combination of advanced lecture and 
laboratory . 

3) Questionnaire procedure : I-Tcores were obtained 
by presenting a list of teaching behaviors with instructions. 
The general instructions are reproduced in Appendix A. All 
responses were made on an Opt-Scan response sheet and analyzed 
via computer. 

h) Results: I-score responses are shown in 
Appendix B on the line marked "OEA." Almost all thirty be- 
haviors revealed variable I-scores to some degree. In Items 
11 and 29, however, I-scores were virtually unanimous. 

Questionnaire procedures and directions proved satis- 
factory. Student reactions to individual items were favorable 
except with regard to Item 2k (should be neatly dressed), where 
9 out of U7 student.-j objected strongly enough to omit a respons 
to tne question. 
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ProcAclur e ana re su it a of 3hA ,XiygA-fiiMZ..i^J^-U9J^ 

1) '" RatlonaXo ; The intcntirn of OEI war, to present 
the 30 tenchinS" behaviors in' several classes » to collect 
I-scores baaed upon the behaviors, antl to obtain the overall 
evaluation of tho instructor (E,j-score) . The objectives were 
(1) to ascertain the stability o,f ..I-scoros as the rsubject 
natter and ai^bject level changed, (2) to test the independence 
of I-scores with the ET-score, through correlation, and (3) to 
ascertain whether presentation order in the questionnaire could 
affect I-scores. 

2) S ample characteristics. ; The student sample in 
OEI consisted ©"f 12^ ' 'Hof stra !Jnivers"ity undergraduates in one 
of four classroom situations: (l) an introductory Art History 
class, (2) an advanced Art History class, (3) an introductory 
Math class or (1+) an advanced Math class. Booh levels of each 
subject area were taught by the same instructor. The subjects 
above were chosen for their dissimilarity in philosophy and in 

approach. 

3) ' - lestionnaire procedur e; I-scores in OEI were 
obtained by presenting the same list of teaching behavior as 
used in OEA, with the same directions and response format 
(Appendix A). ET-scores were also obtained (See Appendix C). 

Each class group was divided randomly into three 
order groups, by dividing tht 30 behaviors into sets (designed 
A, B, and C) of 10 each, and assigning one of three orders 
(ABC, BCA, CAB) of the sets to each order group. 

U) Data analysis ; Differences in mean I-score due 
to classroom situation for each of 30 teaching behaviors were 
tested via a 2X2 analysis of variance (ANOVA) procedure, with 
two levels of subject matter iMath vs. Art History) and two 
levels of subject level ( Introductory vs. Advanced). To 
deter-nine I-score vs.. E^^-score independence, correlational 
tests were performed. Chi-square analysis investigated the 
relationship of three questionnaire orders and the I-score 
response distribution for each item. 

5) Re8ult_s ; I-score responses are shown in 
Appendix B on the line marked "OEI." As in OEA, almost all 
thirty behaviors revealed variable I-scores to so..e degree. 
Discussion of I-score distributions in OEI vs. I-score dis- 
tributions in OEA will be deferred \'.ntil Section H. 

With regard to the inflaenceo of subject area, and 
subject level on I-scores of the 30 teaching behaviors studied, 
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the analysis fi-eoultB is shown in Appendix I). Subject ui'ea 
as a variable- influenced only I-oeorea for items 3, T, 8, 2? 
and 30. Subject a^Vel as a variable influenced only I-scores 
for Item '22, The interaction of Area and J^evel influenced 
only I-scores of Items 5, 13 and 25. In all, only nine F-testa 
out of ninety performed (10^) were significant at the .05 l«vol. 
Since by chance alone, ^% would have been significant, it is 
concluded that the 9 significant tests arc not reliable enough 
for a detailed examination hej*e. Tentatively, one may con- 
clude that subject .level and subject area, as dissimilar as 
Math and Art History, have only minimal effects on I-scores 
for t\--^ :^0 behavior'^ --^ led here. 

(.'orrel.ix-U. ■ *h In OSI iw • ;. . .lu of thej 30 

I"f3cores 00 the hrji-s^jore reve Ueu sf,v^\. U' I sif^ni t^icant as 

socia^.-ioAS (Items I, - 12, 13, 15, and Tij). '^.^nce 23% 

(T/30; was not signi i ica .t ly different from "^X c^-Uc'.oo frc'. 

„ chai^gs U-te U for pr c k ov ; 2 ons ) , further di cuc -.lori oi the 
T;*;rti. Milar si^ni f I r.nnt I- ^ioorea does not, ---..fc . <■.. arrant e-" t're- 
pc ix-. d ill App-;. i:.ix a v. ...t .-VJl) 

Ar ^' r- ! 1; ■ f ^ ■■ : tests ■ 'Vi-ences in I-sco - '^ 
for f•?■.c^ of thirty its.- ■.. l\. . 1 1- -i; u t: ••; f ? <-v--ffrov. p reveal. i3 < •■• 
s £ir.f.ie :;• 1,- . j :■ -.1 r. • r. -::n.x <^.\nr I \ ). Since 1 or 

.significant or dtr -g~: cvp eii-. cca : „Ia n ■ tceA pr.=!dicted b\ 
chance alone (given the .05 level of signif icancp ) from F-teets 
performed, the particular significance cited here was ignore''., 

Proceaur • an ' re sv It a of t^^ bf. ond study {. Cod t : ■ 

^iJ^lj^!P!r;9. Since OIL 7 -a OEI had laid the f.i.-'. 
work for accepting the "l-score as a somewhet reliable measure,, 
the intention of OEIT v^g to present ?<: + caching behaviors in 
each of two individual Questionnaire forLis. administered 
simultaneously) and obtain, not only 1-scores, but E, 0, D, S, 
and Eij-scores as well. In addition, demographic information 
of the student was obtained for later analysis (Appendix F). 

Sample char acte ristics ; The student sample in 
OEII con.«3isted of 132. Nassau Community College undergraduates 
all having in one of five sections the same Psychology instruc- 
tor, with 65 having one set of 10 behaviors and 6? having the 
other set of 10 behaviors. Questionnaire administration time 
necessitated the division of test items to sets of 10. The two 
groups were relatively equivalent in sex, school class, major/ 
nonmajor distintion, and expected grade from the course. 

In the results to be reported below in OEII, the 20 
items will be analyzed ati a single group, regardless of which 
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Get of 10 each item belonged. 

3 ) (j,u e a 1, 0 n n ft 1 r e p r ,9 ffi,e (i,u r .e : - = B e h a v i o r n in 0 K 1 1 
were taken from earlier studied onen in OV*k and OKI {Itemn 
I, 2, 5. 7v 8, 10, 12, l3, Ik, 15, 16, 18, 19, 21, 22, , 
27, 28, 29y and 30). TheSnstruc t ions rcjgarding l-scoreis, 
E-scoren, 0-scores, and S-score& are reproduced in Appendix 
G, H, and J respectively* The general directions are re- 
produced in Appendix Wording of questions for OEII and 
0EII3? studies are shown in Appendix L.' 

D-scores were computed from the X-scores and 0- 
scores of each item. 

k) Data analysis ; In OEII, the issue of inde- 
pendence of I-scores, 0-score.s, D-scores, and E-scores, from 
the Pi'p-Bcore was engaged via correlational tests. In addition, 
the possible influence of the student's sex and anticipated 
grade upon the above scores was tested via a 2X3 AIIOVA design, 
with two levels of sex (male or female) and three levels of 
anticipated grade (A, B, C or lower). 

5) Results ; With regard to I-scores in OEII, cor- 
relations with E^-scores revealed significant results in only 
two of the possible twenty items (items 12 and 25) as depicted 
in Appendix S. The distributions themselves are reported in 

Appendix B on the line marked "OEII." The ANOVA results on 
the demographic effects are seen in Appendix M, showing one 

significant effect (an interaction of sex and grade) in Item 
18, out of a possible sixty tests. The minimal effects upon 
I-scores adds a substantial degree of confirmation earlier 
tests of stability performed on GEA and OEI . 

In contrast, with respect to E-scores in OEII, 
significant correlations with E^-scores, depicted in Appendix 
CL, were frequently observed. Eighteen items showed a signifi- 
cant relationship between Erji-score and E-score, In only two 
cases, Item 13 and 25> was independence indicated. The ANOVA 
results on the demographic effects of sex and anticipated grade 
are seen in Appendix M, Significance was seen in six items out 
of the twenty (items 5, 10, lU, 21, 22, and 29). Sex was a 
factor for evaluatively reacting to how the instructor followed 
an outline (item 2l). Anticipated grade influenced the rating 
on stating material (item 5), description of course require- 
ments (item 10), out-goingness (Item ik) , and clarity of 
explanation (item 22), An interaction effect was seen regard- 
ing the instructor's friendliness rating (item 29). 

With respect to 0-scores in OEII, significant cor- 



relations with Kf^i-scores, depicted in Appendix ty were 
observed in twelve out of the twenty items {items 1, 5, 8, 
13» iU, 15, 16, 18, 22, 27, 28, and 30), with coefficients 
ranging from -i-.g? to ^.52. Independence from Eip-score was 
indicated in how frequently the instructor assigned homework 
(item 3), displayed intellectual broadness (item T), des- 
^ cribed course requirements (item 10), was sarcastic (item 12), 

was flexible (item 19), followed an outline (item 21), over:- 
demanded (item 25) and was • unfriendly (item 29). The AKOVA^ 
results for ^the demographic effects are seen in Appendix 
Significance was achieved in six items out of a possible 
twenty. Sex was a factor in estimating how frequently the 
instructor generated excitement (item 8 ) and followed an 
outline (Item 21), Anticipated grade influenced observa- 
tions regarding organization (Item 13) , outgoing behavior 
(Item lU) and level of presentation (Item 30). No interaction 
effects were observed. 

With respect to D-scores in OEII, only eight of the 
correlations with E^-score were significant (depicted in 
Appendix C) . Twelve items were independent of the Erp-score 
with .regard to their D-scores. As will be discussed later, 
the fewer number of significant D-score correlations with 
ET-scores relative to significant E-score correlations may 
bode well for the possible use of D-acores in evaluation. 
The AKOVA tests for D-scores were not performed. 

While an extended survey of the results of OEII will 
be deferred until after the description of OEIII, a few 
statements can be made here. When correlations are compared 
between various scores and the Erp-score, one sees the E-score 
as having the most frequent associations (l8), followed in 
frequency by 0-scores Cl2), and D-scores (8). I-scores car- 
ried the least frequent statistical associations (2) with the 
E^-score, One sees in part a similar relationship in the ANOVA 
results. The F~ scores and 0-scores were relatively more 
affected by the sex of the student and the anticipated grade 
than were I-scores. D-scores were not a part of the analys is 
in OEII. 

G . Procedure and results of the third study (Code; OEIIll 

1) Rationale ; Another study was undertaken prin- 
cipally as a replication and extension of OEII. In OEIII, a 
larger population received a total of 10 teaching behaviors 
and the usual I, E, 0, D, S, and Eip-scores were obtained. 
Demographic influences as well as independence from the E«p- 
score were investigated in OEIII as in OEII. 
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^ ^ .^.^BXii!™^ )^'^v^^cteri j cr> r T h t va1 rn t r. am p 1 e in 
II consisted of I"l Nnss^at) ConmuiUty Co]\rirc \n;.i r r ^ ^1 u r t . 
having in one of five flections the ^^atMo T r*.y v' !io U^*: \ In- 



B tyuc tor • 

^ Ll]i?Jj^A£5A^LL^^Il^^ i5ehaviors in OKIli 

vel'e taken from one of the two f^jets studied in OKTI. Tl\ls 
included the ten behaviors. of Itemr, B, 12, lU , 15, 19, 21, 

27 , 29 J and 30 (Appendix ft) • 'The sane procedure and in- 
uctions were used in OKIII as was us on in OEII (Appendix j 
ough K ) . 



t h 



0 

f 0 
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Res ults : Ihe d i s t r i bu t i ono c j -scores 
HI are described in Appendix B, on the* line marked "OEIII, 
the ten behaviors studied. Only one correlation of an 



score with the Erp-score (Item 21) was observed (Appendix &) . 
nee zero or one correlation out of 10 might result from 
cxance alone, this sin^rle correlation was ignored. The ANOVA 
t(:sts for influence of sex. and anticipated grade on I~score 
sitowed significance in Item 8 (generating;; excitement) and 27 
V 'ip- to-dva t e competence), (Appendix M), 



With respect to E-scores in 021X1, signif 
relations with t'ne t.^-s^ore were observe-i in eight 
items (Items 0, 1I4 , 15, 19, 21, 27 , 29, and 30), as 
i/n Appendix C. Item 12 (related to the instructor^ 
lu 25 ( over-denandedness ) had E-scores which were 
tlie iJm-score, The ANOVA results on demographic 
:i .^KIII revealed only Item 27 having any sex^ anti 
i --^^^act ion effect (Appendix M) , out of 30 F-tes 
[\4;ain, due to possible chance effects, the single s 
|was ignored. 



icant cor- 
of the ten 
depicted 

s sarcasm) 
i ndependent 
inf luenc es 
c i pat ed grade 
t s performed • 
ignif icance 



V/ith respect to 0-scores in OEIII, significant cor- 
relations with the K»i'-score were observed in eight of the ten 
,items (Items S, 12, lU , 15, I9, 27 , 29 , and 30) as depicted 
in Appendix B. Items 21 (related to following an outline) and 
25 (related to over-demanding) had 0-scores independent of the 
Lrn-score. The ANOVA results (Appendix M) revealed none of the 
10 items having 0-scores that were significantly influenced by 
either sex or anticipated grade. 

With respect to D-scores in OEIII, significant cor- 
relations with the Erp-score were observed in seven of the ten 
items (Items 3 , 12, 15, 19 , 27 , 29 , and 30), as depicted in 
Appendix G. Items ih (related to outgoingnes s ) , 21 (related 
to following an outline) and 25 (related to over-demanding) 
had D-scores independent of the Erp-score, The AKOVA results 
for demographic influences (Appendix M) revealed that sex was 
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a uiijnificant factor for D-scores on Item '8 (generating ex- 
citement) and 19 (flexibility) and an interaction effect of 
sex and anticipated .grade was seen in the l)-scores for Item 
19 (flexibility). 

While an extended survey of the results of OEIII 
will be deferred to a later section (Section I), a few state- 
ments can be made here. When correlations are compared be- 
tween varipub scores and the Kij-score, one sees that, as in 
OKII, the I-scores carriea the least frequent statistical 
associations (l out of 10) with the Erp-score. Conparlsons 
regarding the other scores are less clear-cut in OK III than 
in 0£I1. In Olilli, it was seen that iJ-scores haa 0 out of 
10 possible associations significant at the .05 level, 0- 
scores alsp having: 8 out of 10, and D--scores havin^^ 7 out of 
LO. A cle^arer picture, however, will emerge in Section I 
when OEII and OEIII results are directly compared. 

H • i -score distributions and characteristics in PEA , 
OLI, OKII and OEl'ri; 



The distributions of I-scores from as many as four 
studies (ten behaviors were analyzed in OEA, OEI , OEL". , OEII, 
and OEIII) are seen in Appendix B. Since the student sample 
of OEA and OEI differed substantially from that of OEII and 
OEIII, it seems most reasonable to look at the latter two sets 
of distributions a? a test for I-score distribution reliability 
aside from possible changes due to population differences. 
Appendix K abstracts from Appendix B those ten behaviors studied 
:.r, both OEII and OEIII. While Chi-Square tests were not per- 
formed on the pairs of distributions, each member of the pair 
matches the other to a substantial degree. 

Correlational tests of I-scores with the Er^-score 
in OEI, OEII and OEIII are summarized in Appendix 0. It is 
clear that correlations of I-score and Er^-scores are pre- 
dominantly nonsignificant. It seems also instructive to point 
out that none of the 10 significant correlations of any one 
study was replicated. One may then conclude that I-scores are 
independent of the over-all evaluation of an instructor. 

The ANOVA tests of demographic influence in OEII 

aid OEIII are summarized, again in a box-score format, in 
Appendix 0. Clearly sex and anticipated grade have next to 

no influence on I-scores. 

I • Comp arisons of 1> D, 0 , and D-score correlations with 
t he E^-score in OEII and OEIII : 



In the results sections of OEII and OEIII, it was 



i 



noted that the comparable extent of statistical association 
with Erji-scoi'^es for I, 0, and D-scores vas not , cohb ist cut 
as one analy7.ecl OEII v£. OEIII data (Appendix E). To obtain 
a clearer picture, it vas first noted that for oome of the 10 
Ltens in both OEII and OEIII questionnaires, a si^^nificant re- 
sult in OEII was not observed in OEIII and, of course, vice 
versa. 

J-peci fie ally, consistent results were obtained for 
T out of 10 I v_£. Erp correlations 5 T out of 10 0 vj3^. Erj. 
correlations, 9 out of 10 E vs_. E.p correlations, and 7 out 
of 10 D v^. correlations. From those' correlations that 
^ave consistent answers (e.g., were replicated) regarding sip-- 
nificance vs^. nonsignificance, a conservative comparison of 
I, K, 0, and D-scores can be made. 

Significant (at the ^05 level) associations with the, 
.•;.j)-score were obtained for 0 out of 7 I vs_. Etp . cor relations , 
o out of 9 E Vj5^. Erp correlations, 5 out of 7 0 vs. E.j cor- 
relations and k out of 7 D v^. Eij. correlations. It is note- 
worthy that the E-score has a substantially greater association 
with the Eij-score than has the D-score. It is also readily 
apparent that the I-score has virtually no association with the 
i'.l-jicore. 

J . Relationship of D-^scores and E-scores in OEII and 

•/•nlll: 

In the proposal (Pp. ^-8), the concept of discrepancy 
■•*.lrmtion was proposed as a model of evaluation in teacher 
rating situations. Although mention was made there of the 
C. il . Coombs ideas about discrepancy scores (D-scores in this 
project), it seems advantageous to restate some of his theore- 
tical predictions and those of some competing conceptions. 

1 ) Theoretical models and their ^predict ions ; 
According to a theory^ of cog-nitive discrepancy, the I-score 
point represents a reference point from which evaluational 
statements are derived. The theory asserts that a student* s 
evaluation of some teaching attribute is formed by estimating 
the discrepancy between his ideal for behavior related to that 

attribute and what the student sees his teacher do. When the 
real-ideal discrepancy, or absolute difference between his 
ideal point and his observation point, is small, the teacher 
gets a high rating ("superior," "excellent") and if the 
teacher's behavior is far from the student's ideal, the dis- 
crepancy is large and the teacher sets a low evaluation 
(''poor," "incompetent"). The formula for a discrepancy theory 
(Formula 1) is the following: 
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E-scorejj s jo-score^ - I-scorej^ 
X « particular teaching attribute or \)ehavior. 



The representation of the discrepancy formula on a rating 
scale is given in Figure 1, where it is seen that Student A 
has evaluated the instructor more favorably than has Student 
B. 

The discrepancy analysis also asserts that tvo 
people, both giving a particularly bad rating ("poor") of an ^ 
instructor, may have esti^^ijiat ed two different discrepancies ^ 
if the two raters started out with different reference points 
or I-scores. In Figure 2, Student A has his I-score in the 
middle of the distribution of possible ideals. When a dis- 
crepancy between and 0;^ is calculated, the distance is the 
maximum that can be derived given the distribution of possible 
■observations and the particular I of Student A. On the other 
hand, student B with his I-score to the extreme end of the 
distribution of possible ideals , in Figure 2,' would perceive 
a discrepancy between Ig and Og as the maximum that could be 
derived given the distribution of possible observations and 
his particular I (that of Student B). Since both students 
A and B perceived maximum discrepancies from their respective 
I-points, both students would rate (E-score) the instructor 
as "poor." In other words, seemingly identical E-scores 
would be produced even though they reflect very different, 
0 vs^. I discrepancies (D-scores). 

An empirical test of the discrepancy theory described 
above is twofold. First, if the E-score is calculated by the 
rater relative to the rater's personal I-score on some behavior, 
then there should be a significant correlation between the 
D-score and E-score on each behavior for all students, no 
matter where the I-score lies. Since high D-scores imply a 
large discrepancy and high E-scores are measured by the scales 
used in OEII and OEIII (See Appendix H) , the correlations pre- 
dicted by descrepancy theory would be positive in nature. The 
D vs. E-score correlations in OEII and OEIII will be reported 
for all students as well as for students broken down into the 
five possible categories of I-score response. Second, if the 
identical E-scores may indeed be dissimilar discrepancy scores, 
then one may observe average (mean) D-scores, which have a 
common E-score, varying systematically by virtue of the I-score 
of the rater. The mean D-scores of every behavior in OEII 
and OEIII will be reported for all categories of E-score and 
all categories of I-score. 

An alternative explanation for the original student 
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Figure 1. A hypothetical scale-rating of one teacher by 
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Figure 2. Hypothetical distribution of ideals and 
observations . 



evaluations, in direct opposition to real-ideal discrepancy 
theory, is the assertion that R-scores are not derived from 
a.personal ideal but rather derived for all raters from a 
common reference point. This common reference point would be 
the extreme end point of the frequency scale. The formula 
for the non-discrepancy theory (Formula 2) is the following 

E-scorejj s | O-score^j - k| 

X s particular teaching attribute or behavior. 
K = extreme end of the frequency continuum. 



The contrast between two theories of student evalua- 
tions is apparent in Figure 1, where it is seen that, if 
Formula 1 were true, the E-score from Student A would be 
better than the E-score from Student B, but that, if Formula 
2 were true, the E-score from Student B would be better than 
Student A. 

In analogy to the discussion above, an empirical 
test for non-discrepancy theory would be performed by look- 
ing at D-score vs_. E-score correlations and by looking at mean 
D-scores. Specifically, if Formula 2 were true, then the E- 
scores should correlate with D-scores only when the I-score 
is at the extreme end of the frequency score and no correla- 
tion should be seen for the other I-score categories. How- 
ever, the E-scores should correlate with O-scores for all 
I-score categories since K in Formula 2 functions essentially 
as a constant for all raters. Mean D-scores associated with 
a particular E-score should not change in any systematic way 
as a function of I-score categories, according to non-discre- 
pancy theory. 

2 ) Correlational tests of D and E-scores in OEII ; 
As is shown in Appendix P, correlational tests for ten behaviors 
studied in OEII reveal overall associations between D-scores and 
E-scores in 5 items (items 8, 15, 21, 27, and 30). Four of 
these items (item 8, 15, 27, and 30) also had significant D vs. 
E correlations when considering only the first l-score cate- 
gories (the extreme frequency point on the scale). Correla- 
tional values in these items for those in the extreme l-score 
category ranged from +.65 to +.88. Importantly, there were few 
significant D vs_. E correlations in I-score categories other 
than the extreme one. Item 15 is a good example of a high cor- 
relation overall (r=+.lj3), but attributed predominantly to a 
very high correlation in only one I-score category (r=+.88). 
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3 ) Mean D^score aualyglg over I^-score categ.Qrlen 
In OEII : Appendix R shows the mean values of D-*8Core ror ten 
behaviors in OEII, broken down to all possible E-score and 
I-score categories (25 possible combinations). First of all, 
one can see that for those items in OEII for which there is 
a significant overall correlation between D-score and E-'Score 
(items 8, 15, 21, 27^ and 30), the mean D-scores f or ^ each 

E-score category change systematically, as would be logically 
predicted. It is also apparent that it is not easy to dis- 
cern a systematic variation of mean D-score in any E-score 
category as one varies the I-score* Discrepancy theory would 
say that mean D-scores which share a common E-scgre should be 
smaller for middle-of-the-ecale I-score categories , than for 
extreme-end I-scale categories. 

In a few cases, one observes patterns predicted by 
discrepancy theory . In E- score category 3 for Items 8 , 15 » 19 f 
2l, and 30, one observes a pattern of decreasing D-scores as 
from the extreme to middle I-score categories. However, two 
points must be made regarding any strong conclusions. First, 
the M values of each category are very small; and, second, the 
patterns that are observed are relatively infrequent. If 
discrepancy theory were a pervasive fact of life then one would 
expect to see more such patterns than actually were seen. The 
n^xt study, OEIII, with an increased N, hopefully, would pro- 
duce more persuasive information, 

k ) Correlational tests of 0 and E-scores in OEII : 
Appendix Q shows that all nine analyzed items in OEII show 
overall associations between O-scores and D-scores, It is 
also apparent that the high correlation maintains itself for 
a'^reater number of I-score categories in OEII than was observed 
in the earlier section on D vs. E-score correlations. Item lU 
is an example of an overall 0 v£, E correlation of +,76 being 
also observed for I-score categories 1, 2, and 3. This finding 
casts considerable doubt on the validity of Formula 1, as dis- 
cussed earlier, and in turn the validity of discrepancy theory. 

5) Correlational tests of D and E-scores in OEIII : 
As seen in Appendix P, overall correlations between D-scores 
and E-scores in OEIII were significant in six of the 9 items 
analyzed in OEIII (items 8, Hi, 15, 19» 27, and 30), It should 
be pointed out that only four items (Items 8, 15, 27, and 30) 
showed significant overall correlations in both OEII and OFIII. 
As was seen in OEII, the D vs. E correlations in OEIII were sig- 
nificant only when the I-score category was on the extreme end 
of the frequency scale, 

6 ) Mean D-score analysis over I-score categories 

in OEIII : One of the intentions of OEIII was to expand the N 
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values from which mean D-scores could be more easily observed. 
Appendix o shows the mean D-scores for combinations of E-score 
and I-score in the 10 items in OEIII. As in OEII, there are 
a few items which show decreasing mean D-score values as I- 
acores go from extreme to middle (under ^l-score category 3 
for Items 8 12, lU, 19, 2l). Items 8, 12, 19. and 21 contain 
patterns in both OEII and OEIII which are predicted by discre- 
pancy theory. Again, however, it is equally easy to see 
patterns of D-score values which are not predicted by discre- 
pancy theory. Consequently, the information even in OEIII with 
the increased N does not show clear evidence for the lorlcal 
implications of Formula 1. 

Nonrelational t ests of 0 and E-scores in OEIII- 
Appendix Q reveals that the 0 vs. E correlations in OETTTTe- 
fi np??T*^r^%^^"*^^\^^'''^^''Ss of OEII. Nine of the ten items 
in OEIII showed overall correlations between 0-scores and E- 
scores and some degree of maintenance of that correlation when 
broken down into I-score categories. Item ll*, as an example 
has an overall correlation of *.62 which in llscore categories 
1, 2 and 3 is +.57, +.61, +.63 respectively. 

, ^ Conclusions; The empirical tests reported above . 

m Section J favor a non-discrepancy theory of student eval- 
uations. In turn, they cast serious doubt on the Coombsian 
discrepancy theory which was presented in. the form of Formula 
1. A further discussion will follow in Section L of this 
Report, 

^* Predicti ons of an aggregate of E-scores ; 

In the Proposal for this program of research, a good deal 
""^^ possible prediction of an overall 

evaluation (ET-score) by virtue of combining (via a given 

I^Sas au?^?a?!5%r?^''^*^''!!? °^ specific teacher behaviors. 
It was suggested that a combination of one rater's D-scores 

^nr^lriJf.f^'^^S^^^^r^?'''' ''^^^^ provide the better means 
for predicting that rater's E^-score than an equivalent com- 
bination of E-score; The reasoning behind this assertion lies 
in the purported superiority of the D-score over the E-score 
as being more valid an indicator of a studen?'rperception of 
a teacher. A sideline interest in aggregations using the 
S-score as a weighting factor. Presumably, the S-score would 
indicate how salient .a given behavior is in the rater's eyes 
of thrteacheJ*"* ^^^^ behavior would be in a general evaluation 

, Procedure in OEIII: Each of the 123 student raters 

in OEIII gave D-scores for 10 behaviors. E-scores for 10 be- 
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liavlors, and a J3inr:i«- P,p-Bi;oro. For oach rater, the mean 
D-oeore was calculated over the XO bohavlnr»j and the mean 
K-score vas calculated over tho same* 10 btihaviora. For each 
rater, the mean U-scoro and the mean K-score was correlated 
with that rater's Kn'-score. 

I 

In addition, fo,r each rater, the mean D-score and " 
me&n E-scorfe was calculated over the 10 behaviors, using as 
a multiplicative weighting factor the S-score for each be- 
haviore Since the highest salience score was 5 (in a .range 
from Ito 5), no transformation of S-scores was necessary. 
The formulas for the weighted average of each rater were as 
follows : 

Ig 10 
R«E or S»P o ^ Si D 

i=l ial 

i = one of the ten beha'rlors 
S^a the S-score for behavior i 
Ei« the E-score for behavior i 
Di= the D-Bcore for behavior i 

Values obtained for each of 123 raters were then correlated to 
Eip-score of each rater. 

2) Results in OEIII ; The findings from this analysis 
is summarized in Appendix T. The correlations between mean 
E-scores and Eij-scores, and between mean D-scores and E^-scores 
were virtually identical (r=+.56 vs,. ra+.58) and not impressively 
high. The correlation of D-scores and E-scores from single 
behaviors vs. ET-scores were often of this magnitude or higher 
( See Appendix E ) . 

Similar correlations between S*E and Eiy-scores , and 
between S*D and Efp-scores, produced little or no improvement. 
Appendix T shows that now the correlations were +.55 and +.57, 
respectively. 

Since the aggregation formulas were so inadequate, no 
other analyses were made. The attention was focussed on the 
issues of discrepancy theory, discussed in Section J. 

L . G eneral summary, conplusipns and future ^lans : 

While rather specific interpretations of the data col- 
lected are made in earlier sections of this Final Report, some 
major conclusions should be made here as the report draws to an end 
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1) Regarding the characteristics of I-acores, one 
can well say from all four studies that there i» sufficient 
variability in I-score response for one to dispel the appeal- 

• inig hypothesis that an educator can decide a single ideal fre- 
quency of behavior and assume a correspondence to the view of 
tne students. In addition. Appendix B shows that many teaching 
behaviors do not show an extreme frequency as the modal ideal 
response. The appearance of non-monotonicity in 1-score clearly 
makes difficult any simple transformation of observation scores 
into an evaluative statement regarding teaching behavior. 

2) While I-scores are variable, the data shows that 
the distribution of I-scores are rather stable with regard to 
the sex of the student, the anticipated grade of the student, 
tbe course content, the course level, or the actual overall 
evaluation of the teacher (Appendix D, E, aad M). A comparison 

of I-score distributions from OEII and OEIII in Appendix N reveals 
a kind cf reliability of I-score response over differences in 
teacher and student population. 

The stability seen in I-score distributions seen in 
this project does not, however, ensure stability from other 
sources. For example, it is possible that I-score distributions 
may vary as a result of differences in classroom procedure, e.g., 
lecture vs. discussion. Future studies should investigate the 
importance of teaching style, as well as course content or course 
level of instruction, 

3) The heart of the program was an attempt to validate 
the assertion that evaluational responses were the result of 
cognitive discrepancy calculations between a perceived ideal fre- 
quency and a perceived observed. The conclusion drawn from the 
data is that an evaluational statement, as in an E-score, is not 
constructed from a discrepancy between that student's O-score 
ond his I-score, but rather more likely constructed from a dis- 
crepancy between that student's 0-score and the extreme e n d of 
the I-score scale. The calculation of mean D-scores in Appendix 
R and S show little or no changes as the I-score (supposed 
reference point) varies. 

The result appears paradoxical. I-scores show vari-- 
ability and, from the student's report, are sincere and meaning- 
ful. Yet, the I-score apparently does not play a role in the 
creation of an evaluation of the teacher's behavior. A resolution 
of this paradox may come from two avenues of future experimenta- 
tion. One avenue can be termed the Proximity Problem; the other 
avenue can be termed the Semantic Problem. 

The Proximity Problem involves basic structure of the 
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questionnaire. The first pftp.e elicited I-scoreR; the necond 
page, R-acoresi and the third page, 0-scoree. While it was 
deliberately stated in the instructions that the I-scoros bo 
related to the specific course and situation, it is possible 
that answering the I-score section was done on an abstract level, 
without relating notions about I-scores to actual evaluative 
feelings. If the I-score and the 0-score were more proximal in 
spatial terms, the derived D-score might be more closely related 
to evaluative feelings about a particular behavior of the 
teacher. A study, planned for the summer of 197^ » will vary 
the ordinal relationships of the I-score, E-score and 0-score 
questions and also vary the proximity between the I-score and 
0-score portions of the questionnaire. If the non-participa- 
tion of I-scores in evaluation were due to spatial factors, then 
as proximity of I-score and 0-score increased, so would the 
importance of the I-score in the calculation of teacher evalua- 
tions. 

The Semantic Problem involves the necessity in the 
studies reported here of re-wording a teacher behavior (where 
scales of I-score and O-score could be constructed) into a 
teacher attribute (where scales of E-score could be constructed.) 
As can be seen in Appendix L, in some instances the transfer 
could be made easily (e.g., "Was (should be) flexible" to 
"Flexibility"), In some instances, the transfer could have 
created semantic error (e.g., "Asked more than students could 
do" to "Burden of assignments"). An improvement might be seen 
by presenting the evaluational question not as an attribute 
with a scale from "outstanding" to "poor" but rather as a 
behavior with a scale from "very satisfied" to "very dissatis- 
fied." While the twc formats of an evaluation may be closely 
related, the latter format may prove more understandable to a 
student respondent. It is possible that with semantic error 
reduced, the role of the I-score in evaluation may be observed. 
A study, also planned for the summer of 197^, will compare the 
format styles of E-scores as described above. 

If the . proximity of I-score and O-score questions 
were maximized and if the potential semantic error in behavior 
vs. attribute wording were minimized, and if I-scores were still 
seen as not' playing a role in the evaluational process, then 
there would be little served by incorporating student ideals in 
teacher evaluation scales. The concept, however, of student 
perceptions of teaching ideals appears too valuable from a 
sociological view for an easy dismissal, without additional 
analysis and experimentation. The present research can be viewed 
as a cautionary position regarding discrepancy theory, not as a 
final refutation. 



/ 
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Appendix A 

General Instruction Page for Q EA and OEI Questionnaires 

This questionnaire should take only 10 or 15 minutes to 
complete. It will be done anonymously therefore do not 
write your name, ID number or any personal identification on 
the machlne-scorable answer sheets which are enclosed with the 
evaluation items. 

We are interested in your honest Judgment of the ideal 
frequency of behavior for the following 30 items. We would like 
to know how frequently you as a student in this p articular class 
w ould like or prefer a given teaching behavior to occur. 

If, however, any of the first 30 items appear objection- 
able or Inapplicable, feel free to omit your response to that 
item. 

Ea"!? item can be answered by marking, any one of five 
possible frequencies associated with that item. Here is a sample. 
item ! 

Should be friendly. 

a. Almost always 

b. Often 

c. Occasionally 

d. Seldom 

e. Almost"" never 

If you feel, for example, that the instructor of a course 
such as this should be friendly or that you would prefer that the 
instructor be friendly "almost always" you should mark with a 
pencil the (a) alternative for this item in your answer sheet. 

REMINDER: We are NOT Interested in how the Instructor ijB, but 

rather in how the instrucioor of this course should be. 

After you ere done with the 30 items, please answer item 
31, the general evaluation question located at the end of the 
item sheet. 
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Item in fttteetionnalre 



Percentage of each 
Reaponee Mode of I-score' 



Study Code 


1 


2 


3 JL 




1. Should demonstrate 


OSA 


80 


15 


0 2 


3 


comprehensive knowledge 


OEI 


Lj 0 


15 


U 1 


0 


of his/her subject. 


OEII 


53 


26 


19 2 


0 




OEIII 




not 


DP ds en't ©d 

JM A w 0 ^ M W w %A 




2. Should exhihit a sense 


OEA 


32 


59 


0 0 


0 


of humor. 


OEI 


22* 




51 1 


0 




OEII 




not 


presented 




• 


OEIII 




n w V 






3. Should assign much 


OEA 


0 


n 

w 






homework. 


OEI 


o 


10 




18 




OEII 


2 


3 


UU 29 


22 




OEIII 




not 


presented 




U. Should try to show practical 


OEA 


60 


36 


U 0 


0 


V applications of the subject 


OEI 


39 


37 


18 2 


1 


matter . 


OEII 




not 


presented 




\ 


OEIII 




not 


presented 




5. Should state material 


OEA 


85 


15 


0 0 


0 


clei^arly . 

i 


OEI 


11 


10 


3 0 


0 


OEII 




T 


3 2 


0 


( 


OEIII 




not 


presented 





6. Should be understanding 
of students. 



OEA 61 36 3 0 

OEI 59 3k 6 1 

OEII not presented 

OEIII not presented 



0 
0 



7. Shoudl display intellectual 
interests in subjects other 
than his/her own 



OEA 


23 


30 


30 7 


0 


OEI 


31 


31 


35 3 


0 


OEII 


25 


25 


U3 6 


2 


OEIII 




not 


presented 





^Response mode "1" ■ "almost always" and response mode "5" 
"almost never." 
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Percentage of each 
Item in ftuestlonnair e Response Mode of I-score 







study Code 


X 


o 




1. 




8. 


Should generate excitement 


OEA 






8 


0 


U 




about subject being taught 


OEI 


59 


33 


6 


2 


0 




A 1? T T 






11 


0 


n 






■ OEIII 


61 


32 


6 


1 


0 


9. 


Should maintain a tight 








U9 


13 


< 




control over class. 


OEI 


11 


?3 




lU 


10 










no b 


presented ^ 








OEIII 




not 


presented 


3 

1 


10. 


Should describe in detail 


OEA 


24 


O A 

39 


28 


7 


I 

d 




what should be done and 


OEI 


33 


30 


22 


12 


■ 3 




how it should be done. 


A ^71 *F *F 

OEII 


3o 


3U 


25 


a? 


X 






OEIII 




not 


presented 




11. 


Should be well prepared 


OEA 


fS ^ 

96 




0 


0 


0 




for class. 


OEI 


83 


lU 


2 


1 


0 






0£XX 




not 


presented 








OEIII 




not 


presented 




12 . 


Should be sarcastic with 


AT? A 


o 
c 


A 


22 


38 






students. 


OEI 


2 


2 


12 


32 


52 






OEII 


1 


2 


23 


32 


ho 






OEIII 


0 


3 


11 


25 


61 


13. 


Should present the course 


OEA 




2? 


3 


3 


A 
U 




in a highly organized 


OEI 


uo 




15 


5 


0 




manner • 


OEII 




31 


22 


7 








OEIII 




not 


presented 




14 . 


Should be dynamic and 


OEA 


55 


3^ 


' 13 


0 


A 
U 




outgoing 


OEI 


17 


C A 

50 


29 


3 


X 




OEII 


36 


UU 


19 


1 








OEIII 


35 


50 


Ik 


1 


" 0 


15. 


Sftould encourage student 


OEA 


Uo 


30 


30 


0 


0 




participation in class. 


OEI 


k2 


37 


17 


3 


1 




OEII 


38 


Ul 


19 


3 


0 






OEIII 




35 


18 


3 


0 
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Percentage of each 
Item In Questionnaire Response Mode of I-Bcore 





Study Code 


1 


2,. 




k 


Should tolerate dls-. 


OEA 


hi 


38 


13 


2 


agreement 


OEI 


63 


26 


10 


2 




OEII 


55 


36 


8 


2 




- OEIII 




not 


presented 



17. Should establish a good 
rapport with students 
In the classroom. 



l8. Should allov students to 
volunteer their own 
opinions, 



19. Should be flexible about 
running the course. 



OEA * 


79 


21 


0 0 


OEI 


62 


31 


5 2 


OEII 




not 


presented 


OEIII 




not 


presented 


OEA 


U8 


33 


15 h 


OEI 


6U 


29 


5 2 


OEII 


82 


16 


0 0 


OEIII 




not 


presented 


OEA 


9 


27 


33 U 


OEI 


Uo 


3U 


20 . U 


OEII 


UO 


Ul 


18 2 


OEIII 


55 


33 


12 0 



20. Should make Isolated 


OEA 


53 


33 


U 0 


facts fit into a whole. 


OEI 


UU 


37 


15 3 




OEII 




not 


prosented 




OEIII 




not 


presented 


21. Should follow an outline 


OEA 


17 


U7 


26 6 


closely. 


OEI 


U 


7 


37 37 




OEII 


11 


3U 


Ul 10 




OEIII 


10 


19 


50 lU 



22. Should rephrase explana- 


OEA 


55 


36 


9 


0 


tions to clarify points. 


OEI 


UO 


35 


22 


2 




OEII 


38 


U2 


19 


0 




OEIII 




not 


presented 



23. Should motivate students 
to work hard. 



OEA 53 28 13 U 

OEI 28 1+5 20 7 

OEII not presented 

CEIII not presented 



22- 



I tem in gueatlcanalre 



SU. Should be neatly dressed 



25* Should ask for more than 
students could do. 



Percentage of each 
Response Mode of I-score 



Study Code 



26. Should try to relate course 
material to that of other 
courses 



OEA 


32 


20 


35 


11 


2 


OEI 


17 


2U 


3U 


16 


8 


OEII 




not 


presented 




OEIII 




not 


presented 




OEA 


2 


U 


15 


U3 


36 


OEI 


3 


6 


30 


20 


Ul 


OEII 


2 


3 


16 


26 


52 


OEIII 


0.. 




15 


22 


59 


OEA 


28 


19 


i*5 


6 


2 


OEI 


19 


2U 


36 


lU 


7 


OEII 




not 


presented 




OEIII 




not 


presented 





27 • Should be up-to-date on 

contemporary issues related 
to the course* 



28, Should become personally 
involved with his/her 
students 



OEA 


70 


26 


2 


0 


2 


OEI 


59 


30 


8 


1 


0 


OEII 


68 


19 


10 


3 


0 


OEIII 


81 


19 


0 


0 


0 


OEA 


15 


23 


32 


26 


\ 


OEI 


11 


20 


36 


20 


13 


OEII 


10 


25 


U5 


12 


9 


OEIII 




not 


presented 





Should he unfriendly and 


OEA 


0 


2 


2 


8 


67 


aloof • 


OEI 


1 


2 


3 


18 


76 




OEII 


1 


1 


3 


1 


93 




OEIII 


1 


0 


2 


U 


93 



30. Should present subject 


OEA 


U5 


36 


15 


2 


2 


matter at a level compre- 


OEI 


52 


23 


18 


6 


1 


hended by all student** In 


OEII 


69 


25 




0 


2 


the class. 


OEIII 


7U 


21 


5 


0 


0 



-23- 



Appendix C 



Overall evaluation question (Eip-score) 
for OEI, OEII and OEIII. 



How would you rate your instructor in general overall 
teaching ability? 

a. An outstanding and stimulating instructor. 

b. A very good instructor. 

c. A good instructor. 

d. An adequate but not stimulating instructor 

e. A poor and inadequate instructor. 



/ 



Appendix D 

■ ANOVA effects on I-scores In OEI 

Questionnaire Subject Area Subject Level Interaction 
Item Effect ■ Effp^t Effect 



3 


• ' P«5.53, p< .05 


N.S. 






N.S. 


5 


N.S. 


N.S . 




F«5. 


99, P<.05 


T 


F«5.38, p < .05 


N.S. 






N.S. 


8 


F«5.22, p < .05 


N.S. 






N.S. 




N.S. 


N.S. 




F=U. 


17, P<.05 


22 


N.S. 


Fa5.8U, p 


<.05 




N.S. 


25 


N.S. 


N.S . 




F-U. 


87, P<.05 


27 


Fs2U.l6, p< .001 


N.S. 






N.S. 


30 


F«T.9l*, P< .01 


N.S . 






N.S. 




Non-siKnif icant Results in 


any of the 


above 


ANOVA 


tests 



Items 1, 2, U, 6, 9, 10, 11, 12, lU, 15, 16, 17, 18, 19, 20, 21, 23, 2U. 
26,28,29. 

Note ; Of a total of 90 ANOVA tests in OEI, nine were 
significant at the .05 Type 1 error level. 
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Appendix E 

Correlation of Eip-score with I,E,0,S, and D-score 

Item in Study 
Questionnaire Code l-'score E-score O-seore S-'SCore D-score 

1 OEI pitisi - - - ■ - 

rB+.58 r«+.30 ^ ^ 
OEII N.S. p».0Ol p=.006 N.S. N.S. 

OEIII not presented 



OEI N.S. 

OEII not presented 

OEIII not presented 

OEI U.S. - - - - 

OEII N.S. psToii ^-S. N.S. N.S. 

OEIII not presented 



OEI N.S. - - . - 

OEII not presented 

OEIII not presented 

OEI N.S. 

ra+.UU r«+.U9 „ „ r=+.6o 

OEII N.S. pr.oOl ps.Oll "'^t pa. 001 

OEIII not presented 

OEI N.S. 

OEII not presented 

OEIII not presented 

OEI N.S. 

OEII N.S. ps*ioi 

OEIII uot presented 



I ♦ 



ERIC 



Correlation of Erp-acore with I,E,0,S, and 13-score 



8 



OEI 
OEII 

OEIII 



N.S. 
N.S. 

N.S. 



Item in Study 
Questionnaire Code V-^Bcore E-score . O^score S-acore 



r=+.52 r=+.43 ra-.27 

pa.OOl p=.001 ps.OlU 

r=+.55 r=+.6l 

p=.001 ps.OOl 



N.S. 



D-score 



rs+.26 
p=.017 

r=+.5U 
pa. 001 



OEI 

OEII 
OEIII 



r=+.27 
ps.OOl 

not presented 
not presented 



10 



OEI 

OEII 
OEIII 



r=+.17 
p=.030 

r=+.28 
p=. 009 
not presented 



N.S. 



N.S. 



N.S. 



N.S. 



11 



OEI N.S. 

OEII not presented 

OEIII not presented 



12 



OSI 
OEII 

OEIII 



N.S. 

r^*,2k ra+.25 

^~.02k p=.021 

N.S. N.S. 



N.S. 



N.S. 



N.S. 



rsT.29 r=+.l6 
ps.OOl pa. 035 



13 



OEI 
OEII 



r=+.23 

pa . OOi* 

U.S. 



N.S . 



OEIII not presented 
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r=+.25 
pa. 021 



N.S. 



N.S. 




Item in Study ^ T^ « 

Questionnaire code I^score E-8Core O-aeore S-.score D-score 



Ik 



OEI 
OEII 

OEIII 



U.S. 

N.S. 



r=-»-.52 r=+.52 r'^'-H v g 
pa. 001 p».001 p«.002 



r=+.60 r«+.60 r=-.2Q „ g 
N-S. pa. 001 p«.001, p«.Oll* 



15 



OEI 



OEII 



OEIII 



r«+.23 
p=.005 

N.S. 
N.S. 



r=+.30 
pa.OOT 

ra+ . 29 
ps.OOl 



r«+.28 
p«.011 

r«+.31 
p=.0Ol 



r«+.25 
N'S. p=.0l8 

r=+.36 
N.S. ps.ool 



16 



OEI N.S. 
OEII N.S. 
OEIII not presented 



r8+.33 
p=.003 



r«+.UT 
ps.OOl 



N.S. 



p=.001 



17 



OEI N.S. 

OEII not presented 

OEIII not presented 



18 



OEI N.S. 

OEII ^ N.S. 

s. 

OEIII not presented 



r=+ . kk 
ps.OOl 



r«+.35 
p=.002 



r=+.31 
N.S. ps.005 



19 



OEI 
OEII 

OEIII 



N.S. 
N.S. 

N.S. 



r=+.32 
ps.OOU 

ra+.38 
ps . 001 



N.S. 

r=+.2T 
ps.OOl 



N.S. 



N.S. 



r=+.21 
N.S. pa. Oil 
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Correlation of Efr-seore with I,E,0,S, and D-score 



Item in Study 

Questionnaire Code I^score E-score 0-soore S^ecore H^acore 



20 



OEI N.S. 

OEII not presented 

OEIII not presented 



21 



OEI 
OEII 

OEIII 



N.S. 

N.S. 

r=+.22 
p=.007 



r=+.22 
p«.033 

ra+.31 
p«.001 



N.S 
N.S 



N.S . 



r«-.?2 
p=.006 



N.S. 



N.S. 



22 



OEI N.S. 
OEII N.S. 
OEIII not presented 



r»+.3T 
ps.OOl 



r«+.35 
p=.0Q2 



r=+. 20 
pe.0U6 



N.S 



23 



OEI 

OEII 
OEIII 



r=+.l8 
ps.021 

not presented 
not presented 



2k 



OEI N.S. 

OEII not presented 

OEIII not presented 



25 



9EI 
OEII 

OEIII 



N.S. 

■ra+.2T 
p=.015 

N.S. 



N.S. 
N.S. 



N.S 
N.S 



N.S . 



N.S. 



N.S 



N.S 



26 



OEI N.S. 

OEII not presented 

OEIII not presented 
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Correlation of Eysscore with I,E,0,S, and D-score 



Item in Study 

Questionnaire Code I-scor e E-score O-ecore S^score D-score 

27 OEI N.S. . - - 

rs+.3T ra+.30 ra-.2U r»+,2U 

OEII N.S. ps.OOl pa. 006 p=.025 p«.02U 

ra+.36 ra+.2T . rs+.lT 

OEIII N.S. ps.oOl p=.001 N.S. p=,028 



28 OEI N.S. - * - - 

OEII N.S. ps.OOl pa. 010 ps.OOl 

OEIII not presented 



29 OEI N.S. - - - - 

OEII N.S. p-.OOU N.S. N.S. N.S. 

r=+.li6 r«-.2T „ ^ r«.20 

OEIII N.S. p^^ooi p,^oo3_ N.S. ^,^012 



30 OEI N.S. - - - - 

r»+.60 ra+.25 r=+.25 

OEII N.S. pa. 0.01 pa. 020 N.S. pa. 020 

OEIII N.S. , 
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Appendix P ' 

( 

Background Questions for OEA« OEI. OKII, and OEIII studies. 



1'. Sex. 

a. Male 

b. Female 

2, Grade point average. 



a. 


3.6 


- If.O 


b. 


3.1 


- 3.5 


c . 


2.6 


- 3.0 


d. 


2.1 


- 2.5 


e . 


2.0 


or below 



3. Year. 

a. Freshman 

b. Sophomore 

c . Junior 

d. Senior 

U. Is this course in the field that you are majoring in? 

a. Yes, this is a course in my field of specialization 

b. No, this is not a course in my field of speciali- 
zation. 

5. Is this course meeting your expectations? Are you pleased 
with your decision to take this course? 

a. Yea . ' 

b. No 

6. Based on your grades in this course so far, what do you 
expect to get as a final grade? 



a . A 

b. B 
c . C 
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* Appendix G 

Instructions for I~acore respotiaes in OEII and OEIII . 

We are interested in your honest Judgment of the ideal 
frequency of teaching Ijehavior for the following 10 items. We 
want to know how frequently you would like or prefer a given 
behavior to occur. Each item can "toe answered by marking any 
one of the five' possible frequencies associated with that item. 

i. 

'.i.' 

? Here is a sample item . 

Should be friendly. 

a. Almost always 

b. Often 

c. Occasionally 

d. Seldom 

e. Almost never 

If you feel, for example, that the instructor should be 
friendly or that you would prefer that the instructor be 
friendly or that you weald prefer that the instructor be 
friendly 'almost always,' you should mark the (a) alt native 
for this item in your answer sheet. Remember: We are not 
interested in how the instructor i_s but rather in how the in- 
structor should be. 
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Appendix H 

Instructions for E-score responses in OEII and OEIII . 

For the following Items we are concerned with an evalua- 
tlonal rating aloag a five-point scale ("outstanding," "good," 
"adequate," "fair," or "poor") of this teacher's competence for 
each given teaching behavior. Each item can he answerea by 
marking any one of the five possible ratings associated with that 
item. Here is a samrle item . 

Kapport with students in class. 

a. Outstanding 

b. Good 

c. Adequate 

d. Pair 

e. Poor 

If you feel, for example, that this teacher's "rapport 
with students in class" was 'good' you should mark the (b) 
alternative for this item in your answer sheet. 
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Appendix I 

Instructions for O-score responses In OEII and OEIII » 

We are Interested In your honest judgment of the observed 
frequency of behavior for the following 10 items. We want to 
know how frequently you have observed a given teaching behavior 
to occur in this class. Each item can be answered by marking 
any one of five possible frequencies associated with that item. 
Here is a sample item . 

Was friendly. 

a. Almost always 

b. Often 

c. Occasionally 

d. Seldom 

e. Almost never 

^ If you feel, for example, that the instructor is friendly 
'almost always' you should mark the (a) alternative for this 
item in your answer sheet. 
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Appendix J 

« 

Instructions for S-score responses in OEII and OEIII . 

We are interested in how important you consider the 
attribute in arriving at a general, over-all view of teaching 
effectiveness. We are not interested in how positive or 
negative the particular attribute is, nor in how the instructor 
ranks with respect to this attribute, only in how much effect 
and observation of this attribut would have on an over-all 
rating of the teacher. Here is a sample item ; 

Spitting on the floor during class. 

a. Of next to no importance 

b. Not very important 

c. Of only moderate importance 

d. Important 

e. Extremely important 

If you feel, for example, that spitting on the floor 
during class would be 'exti-emely important' information in ar- 
riving at an over-all rating of the teacher, you should mark 
the (e) alternative for this item in your answer sheet. 
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Appendix K 

General Instr uc tion Page for OEII and 
OEIII Questionnaires 

This questionnaire is part of a continuing study of 
teacher evaluation processes under the direction of Dr. Charles 
F. Levinthal of the Department of Psychology, and funded ty the 
U.S. Office of Education. 

We are particularly concerned with the various ways of 
looking at teachi ng behaviors . Located at the top of each page 
you will find a set of specific directions indicating what type 
of response is required for the items below. ^ If, however, any 
of the items appear objectionable or inapplicable ,^ you may omit 
your response to that item. 

Each question is numbered and appropriate responses 
should be made by completely blackening the appropriate space 
on the enclosed machin* scorable answer sheet* After you have 
finished answering each page of items go to the next page. Do 
"""t return to make any changes in your answer. 

This questionnaire should take only 20 or 25 minutes to 
complete. It will be done anonymously therefore do not 

write your name or any personal identification on the answer 
shee ' . 
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Appendix L 



Question Statement Formats for Items 
used in both OEII and OEIII 



Item 

Nq» Format for I and 0 scores 

8 (Should) generate excite- 

ment about subject being 
taught . 

12 (Should) be sarcastic. 

Ik (Should) be dynamic and 

outgoing. 

15 (Should) encourage student 

participation in class. 

19 (Should) be flexible about 

running the course 

21 (Should) follow an outline 

closely. 

25 (Should) ask for more than 

students could do» 

27 (Should) be up-to-date on 

contemporary issued related 
to the course. 

29 (Should) be unfriendly ^hd 
aloof . 

30 (Should) present subject 
matter at a level compre- 
hended by all students in 
the class. 



Format for E and S scores 

Generating excitement about \ 
sub j ect . \ 



Sarcasm 



Dynamic and outgoing, 



Encouragement of class 
participation. 

Flexibility about running 
the course. 

Following an outline closely. 



Burden of assignments. 



Competence in recent' ad- 
vances in field. 



Friendliness 



Ability to present subject 
matter comprehensibly to 
total class. 
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Appendix M 



Questionnaire 
Items used in 
either OEII Study 
or OEIII Code 



ANOVA effects u-pon I-scores 



Sex effect 



Anticipated 
grade effect 



Interaction 



8 



OEII 
OEIII 



N.S. 
N.S. 



N.S. 
FaU .61, p < .05 



N.S. 
N.S. 



18 



OEII 
OEIII 



N.S. N.S. 
«• - - not presented - - 



27 



OEII N.S. 
OEIII F=4.8U, p < .05 



N.S. 
N .S. 



N.S 
N.S 



Nonsignificant Results in any of the above ANOVA tests ; 

Items 1,3,5,7,10.12.12,1^,15,16,19,21 ,22,25,28,29,30 

(presented in OEII) 

Items 12,111.15,19,21.25,29,30 

(presented in OEIII) 

Kote: Of a total of 90 ANOVA tests in OEII and OEIII, 
tBree were significant at the .05 Type 1 error level. 



ANOVA effects upon E-scores 



Questionnaire 
Items used In 

either OEII Study Anticipated. 

or OEIII Code Sex effect grade effect Interaction 

t 

5 OEII K.S. PaU.65, P<.05 N,S. 

OEIII - - - not presented - - - 

10 OEII N.S. F«U.25, P< .05 N.S. 

OEIII - - - not presented - - - 



Ih OEII N.S. F=U.l6, p< .05 N.S. 

OEIII N.S. N.S, N.S. 



21 OEII F=6.99. p<.05 N.S. N.S. 
OEIII N.S. K.S. N.S. 

22 OEII N.S. Fa5.8l, p< .01 N.S. 
OEIII - - - not presented - - - 



2T OEII 
OEIII 

29 OEII 
OEIII 



N.S. 
F=5.^3i*, p^ .05 

N.S. 
N.S. 



N.S. 
N.S. 

N.S. 
N.S. 



N.S. 
F»3.Ul, p< .05 

F«3.29, p< .05 
N.S. 



Nonsignificant Results in any of the above ANOVA tests ; 

Items 1,3,7,8,12,13,15,16,18,19,25,28,30 
(presented in OEII) 

Items 8,12,15,19,25,30 
(presented in OEIII) 

Note : Of a total of 90 ANOVA tests in OEII and OEIII, 
eight were significant at the .05 level, and one was 
significant at the .01 level. 
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ANOVA effects upon 0-scores 
Questionnaire . 
Items u . ed in 

either OEII study ,/ Anticipated 

OEIII Code Sex effect grade effect Interaction . 

1 02II N.S. N.S. F=U.OO, p< .05 

OEIII - - - not presented - - - 



8 OEII F=5.03. p<.05 ■ N.S. 



N.S. 



OEIII N.S. N.S. N.S. 



N.S. 



N.S. 



13 OEII N.S. P=3.22, p< .05 

OEIII - - - not presented - - - 

1^ OEII N„S. F»l*.86, p<.d>5 

OEIII N.S. N.S. N.S. 

19 OEII N.S. N.S. P=l*.Ol*, p< .05 

OEIII N.S. N.S. N.S. 

21 OEII F«6.99. p<..05 N.S. N.S. 

OEIII N.S. N.S. N.S. 

30 OEII N.S. F=8.97, p < . 01 N.S. 

OEIII N.S. N.S. N.S. 

Nonsignificant effects in any of the aljove ANOVA tests ; 

Items 3.5,7,10,12,15,16,18,22,25,27,28.29 
(presented in OEII) 

Items 12,15,25,27,29 
(presented in OEIII) 

Note: Of a total of 90 ANOVA tests in OEII and OEIII, 
seven vere significant at the .05 level, and one was 
significant at the .01 level. 



ERIC 



AKOVA effects upon D-scores 



Questionnaire Items 
in OEIII only 



Sex effect 



Anticipated 
grade effect 



Interaction 



8 



FaU.6T, p < .05 



U.S. 



N.S. 



19 



FaU.25, P< .05 



N.S. 



Fa6.92, p< .01 



Nonsignificant results in any of the above ANOVA tests ; 

Items 12, lU, 15, 19, 21, 25, 29, 30 
(presented in OEIII) 

ANOVA of D-scores not performed for OEII data. 

Note ; Of a tota3 of 30 ANOVA tests in OEIII, three 
were significant at the .05 level and one was 
significant at the .01 level. 
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Appendix N 

I 



Item in 
Questionnaire 

8 

12 

Ih 

15 

19 

21 
25 
27 
29 
30 



Percentage of each 

.response mode of I~score 
Study Code i 2 3 Hi 5" 

t 

OEII 65 2U 11 0 0 

OEIII 61 32. 6 1 0 

OEII 1 2 23 32 k2 

0 3 11 25 61 



OEII 36 1+1+ 19 1 

OEIII 35 50 ll 



0 

1 0 



OEII 38 U 19 3 0 

OEIII kk 35 18 3 0 



OEII 1+0 1+1 18 2 0 

OEIII 55 33 12 0 0 



OEII 11 31+ kl 10 3 

OEIII 10 19 50 11+ T 

OEII 2 3 16 18 52 

OEIII 0 1+ 15 22 59 

OEII 68 19 10 3 0 

OEIII 81 19 0 0 



OEII 113 1 

OEIII 10 2 1+ 



OEII 69 25 U 0 

OEIII 7U 21 5 0 



93 
93 



2 
0 
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Appendix 0 



Study Code 



Number of significant correlational tests 
with Em-score (numerator) relative to number 
possible (denominator) in OEII and OEIII 



I-scor e 



E-score 



0-acore 



P- 9 core 



OEI 



7/30 



OEII 
OEIII 



2/30 



1/10 



18/20 
8/10 



12/20 



8/10 



8/20 
7/10 



Number of significant ANOVA tests with 
demographic variables (numerator) relative 
to number poflaihio <;,^^..^«4 n^^t ^.^ | 

S1}ti4Y Cfta^ J-score E-score 0-seore p-gcore 



^/^O 6/60 6/60 not analyzed 

2/30 1/30 0/30 3/30 



-1^3- 
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